Self-contained Entity Discovery from Captioned Videos

This article introduces the task of visual named entity discovery in videos without need for task-specific supervision or external knowledge sources. Assigning specific names to entities (e.g., faces, scenes, objects) video frames is a long-standing challenge. Commonly, this problem addressed as supervised learning objective by manually annotating with labels. To bypass annotation burden setup, several works have investigated utilizing sources such movie databases. While effective, approaches do not work when are provided and can only be applied movies TV series. In work, we take step further propose discover from corresponding captions subtitles. We introduce three-stage method where (i) create bipartite entity-name graphs frame–caption pairs, (ii) find agreements, (iii) refine assignment through entity-level prototype construction. tackle new problem, outline two benchmarks, SC-Friends SC-BBT , based on Friends Big Bang Theory Experiments benchmarks demonstrate ability our approach which belongs face scene, an accuracy close oracle, just multimodal information present videos. Additionally, qualitative examples show potential challenges self-contained any future work. The code data available GitHub. 1